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1. Introduction 


Intelligent Tutoring Systems (ITSs) are sophisticated computer- 
based learning environments (Graesser, McNamara, & VanLehn, 
2005) that often incorporate multiple methods and trajectories 
for interaction based on each user’s unique needs and abilities 
(Jackson & McNamara, 2013; Murray, 1999; Sabourin, Shores, 
Mott, & Lester, 2012; Snow, Jackson, & McNamara, 2014; Snow, 
Likens, Jackson, & McNamara, 2013). Consequentially, students 
often have different experiences and exhibit various levels of con- 
trol during their time within these environments. Such varying 
experiences are often influenced by various individual differences 
(Baker, Corbett, Koedinger, & Wagner, 2004; Baker, Walonoski, 
Heffernan, Roll, Corbett, et al., 2008; Snow, Likens, et al., 2013); 
thus, ITSs provide researchers with a unique opportunity to exam- 
ine how individual differences influence the way in which students 
choose to control their learning experience (Sabourin et al., 2012; 
Snow, Jacovina, Allen, Dai, & McNamara, 2014; Snow, Allen, 
Russell, & McNamara, 2014). 
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When students exert control over their behaviors during learn- 
ing tasks it is often referred to self-regulated learning (SRL). This 
skill has been shown to be an important component of the learning 
process as it has led to positive effects on students’ overall learning 
gains (Butler & Winne, 1995; Harris, Friedlander, Saddler, Frizzelle, 
& Graham, 2005; Jarvela & Jarvenoja, 2011; Pintrich & De Groot, 
1990; Zimmerman, 1990; Zimmerman, 2008; Zimmerman & 
Schunk, 1989, 2001, 2013). Zimmerman (1990) proposed that 
when students take personal responsibility over their scholarship, 
they are more likely to succeed than those students who do not. 
Self-regulated students frequently set goals, plan, organize, self- 
monitor, and self-assess during learning tasks, which allows them 
to remain actively aware of their own actions, knowledge, and 
decisions. 

One characteristic of self-regulating students is their propensity 
to approach learning tasks in a decisive and goal directed manner 
(Zimmerman, 1990, 2008). Recently, researchers have investigated 
this characteristic within the context of ITSs (Hadwin, Nesbit, 
Jamieson-Noel, Code, & Winne, 2007; Sabourin et al., 2012; 
Snow, Jacovina, et al., 2014; Winters, Greene, & Costich, 2008). This 
work has shown that when students plan and exert control over 
their behaviors within a computer-based learning environment 
they perform better compared to those who do not (Hadwin 
et al., 2007; Sabourin et al., 2012; Snow, Jacovina, et al., 2014; 
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Snow, Allen, Jackson, et al., 2014; Snow, Allen, Russell, et al., 2014). 
For instance, Sabourin et al. (2012) examined how students’ behav- 
iors within the immersive game-based environment, Crystal 
Island, related to their use of SRL strategies (e.g., self-monitoring 
and goal setting). Results revealed that students’ with higher levels 
of SRL strategy use were also students who interacted within the 
game-based system in a goal oriented and planned fashion. Simi- 
larly, Snow et al. (2014) examined how students exhibited random 
or deterministic patterns of choice while they engaged within the 
game-based ITS, iSTART-ME. This work showed that when students 
engaged in random interaction patterns within the system inter- 
face, they performed worse than students who demonstrated con- 
trolled interaction patterns. Finally, Hadwin et al. (2007) utilized 
the web-based study software gStudy to examine how patterns 
in students’ study habits related to self-report measures of SRL. 
This work revealed that ordered and goal driven study patterns 
were positively related to SRL abilities. Combined, these studies 
have found that students’ ability to act in a controlled and goal 
directed way is a characteristic of SRL behavior. 

Although self-regulation is crucial for academic success, this 
skill tends to vary widely, as many students struggle to set their 
own learning goals and actively monitor goals during learning 
tasks (Ellis & Zimmerman, 2001). One factor that has been linked 
to variations in students’ ability to self-regulate is prior skill level 
(Kitsantas, Winsler, & Huie, 2008; McClelland et al., 2007; 
Zimmerman, Bandura, & Martinez-Pons, 1992). McClelland et al. 
(2007), for example, examined the relation between regulatory 
behaviors and emergent skills within preschoolers. They found 
that self-regulative behaviors were highly related to the students’ 
scores on an academic aptitude test. Similarly, Zimmerman and 
Martinez-Pons (1986) found that students’ scores on academic 
achievement tests were related to their SRL ability. Thus, higher 
skill levels seem to be related to SRL behaviors. 

However, SRL ability is not static (i.e., unchanging), instead 
researchers have shown this ability is dynamic (Boekaerts, 
Pintrich, & Zeidner, 2000; Hadwin et al., 2007; Zhou, 2013) and 
evolves overtime (Glaser & Brunstein, 2007; Zimmerman, 2008). 
Such work has revealed that self-regulation is not simply 
something students either excel or fail at. Indeed there are many 
factors that can influence the evolution of this skill. For instance, 
Muraven, Baumeister, and Tice (1999) found that metacognitive 
strategy training is effective over time at improving students’ 
self-regulatory behaviors. Similarly, Glaser and Brunstein (2007) 
showed that metacognitive strategy training improved students’ 
SRL abilities. Thus, students who struggle to regulate their 
behaviors are able to improve this skill with the adequate 
instruction. 

Although training has been shown to have positive effects on 
students’ SRL ability, this skill can evolve naturally as well. 
Eshel and Kohavi (2003) demonstrated that students’ ability to 
use SRL strategies improved when they were given high amounts 
of agency over their learning environment. Similarly, Bandura’s 
(1991) Social Cognitive Theory links the process of self-regulation 
to personally agency. Bandura postulated that students who self- 
regulate exhibit reflective and reactive decision-making in their 
choices. Thus, improvements in SRL ability are not just accom- 
plished through external factors such as training. Instead, stu- 
dents must take agency over their actions by deciding how to 
control and regulate their behaviors. Such choices are often reac- 
tionary and therefore evolve overtime as students gain more 
experience and receive increased amounts of feedback from a 
given environment (Bandura, 1991). This work has led to the 
hypothesis that when students are afforded opportunities to exert 
agency over their environment or a given situation, they may nat- 
urally begin to regulate their behaviors without external training 
or prompting. 


This complex interplay between SRL and personal agency is 
especially relevant within the domain of ITSs. As discussed earlier, 
these computer-based learning environments often incorporate 
high levels of agency while presenting students with adaptive con- 
tent as a means to engage and challenge them. Thus, the best indi- 
cation of the evolution of students’ regulatory skills is potentially 
through the examination of their ability to control and regulate 
their behaviors when they are presented with numerous options 
or trajectories. However the evolution of these behavioral changes, 
as can be expected, is difficult to measure and often overlooked 
through the use of traditional self-report measures. Static mea- 
sures of SRL such as self-reports usually focus on students’ memo- 
ries for past behaviors; however, students may not be conscious of 
their changing behaviors. This renders the nuanced and dynamical 
patterns of behavior change hard to measure through self-report 
assessments alone. 

One way to measure the evolution of students’ self-regulated 
behaviors within adaptive environments is through the analysis 
of system log data (Hadwin et al., 2007). Log data (e.g., keystroke, 
mouse click, click stream, or telemetry data) records all student 
interactions within an adaptive environment. Researchers often 
intentionally program computer-based environments to capture 
this information as a means to examine fine-grained interactions 
within the interface. This type of data collection and analysis, 
although tedious, provides researchers with a wealth of informa- 
tion regarding how students choose to exert agency and control 
their behaviors within a system. Log data has been previously used 
to examine how students’ interactions within ITSs influence their 
attitudes (Hadwin et al., 2007; Rai & Beck, 2012; Snow, Jackson, 
Varner, & McNamara, 2013a) and _ performance (Rowe, 
McQuiggan, Robison, & Lester, 2009; Snow, Jackson, Varner, & 
McNamara, 2013b). While informative, these prior studies have 
primarily focused on variations in students’ interaction patterns 
at a coarse grain-size (e.g., frequency of interactions). To investi- 
gate how students exert agency while interacting within an adap- 
tive system, more dynamic and fine-grained analyses that focus on 
the presence of nuanced patterns in students’ behaviors are 
needed. The work presented here combines two dynamic method- 
ologies (i.e., probability and Entropy analyses) to examine how 
individual differences in prior reading ability influence the evolu- 
tion of students’ choice patterns as they manifest over time and 
their subsequent relation to learning outcomes. 


1.1. iSTART-ME 


iSTART (Interactive Strategy Training for Active Reading and 
Thinking) is an intelligent tutoring system designed to provide 
self-explanation and comprehension strategy training to high 
school students (Jackson & McNamara, 2013; McNamara, 
Boonthum, Levinstein, & Millis, 2007). iSTART strategy instruction 
has been shown to be effective at improving students’ comprehen- 
sion and self-explanation ability (Jackson & McNamara, 2013; 
McNamara et al., 2007; O’Reilly, Sinclair, & McNamara, 2004; 
Taylor, O’Reilly, Rowe, & McNamara, 2006). iSTART consists of 
three modules: introduction, demonstration, and practice. Within 
the introduction module, students are provided a brief description 
self-explanation reading strategies. After the introduction module, 
students are transitioned into the demonstration module where 
two pedagogical agents (one teacher and one student) demonstrate 
how to apply the self-explanation strategies to example science 
texts. Finally, after students complete the demonstration module 
they are transitioned into the practice environment where they 
self-explain various target sentences from an example science text. 
The practice module is designed to provide students with the 
opportunity to apply the information that they learned within 
the introduction and demonstration modules.iSTART-ME (Motiva- 
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tionally Enhanced) is a game-based version of iSTART that provides 
students with strategy training within the context of an interactive 
game-based environment (Jackson, Boonthum, & McNamara, 2009; 
Jackson, Dempsey, & McNamara, 2010; Jackson & McNamara, 
2013). iSTART-ME is similar to iSTART with the exception that after 
the practice module students are transitioned into an extended 
practice menu that uses game-based principles and features to 
enhance students’ motivation, engagement, persistence, and learn- 
ing throughout long-term practice (Cordova & Lepper, 1996; 
McNamara, Jackson, & Graesser, 2010; Ricci, Salas, & Cannon- 
Bowers, 1996). Students control their iSTART-ME extended prac- 
tice experience through a game-based selection menu (see 
Fig. 1). This menu was designed to enhance students’ feelings of 
agency over the environment by affording the opportunity to 
choose what to do. 

Within the iSTART-ME extended practice selection menu, stu- 
dents can choose to interact with four types of game-based fea- 
tures (i.e., generative practice games, identification mini-games, 
personalizable features, and achievement screens). Generative 
practice games are designed to provide students with the opportu- 
nity to practice generating their own self-explanations within the 
context of a game narrative. In these games, students are shown 
a text and then are asked to generate a self-explanation of the text 
that they had just seen. Students’ generated self-explanations are 
assessed by the iSTART algorithm, which combines latent semantic 
analysis (LSA; Landauer, McNamara, Dennis, & Kintsch, 2007) and 
word-based measures. This algorithm yields scores on a range of 
0-3. Self-explanations are assigned a score of “0” when they are 
composed of irrelevant information. A score of “1” is assigned 
when students’ self-explanations relate to the target sentence 
but do not elaborate upon any of the given information. A score 
of “2” is assigned to self-explanations that incorporate information 
outside of the target sentence. A score of “3” is assigned to self- 
explanations that incorporate information about the target sen- 
tence from their prior knowledge. This algorithm has been shown 
to score self-explanations comparable to expert human raters 
(Jackson, Guess, & McNamara, 2009; McNamara et al., 2007). 

Identification mini-games are designed to provide students 
with a different type of strategy instruction, mainly, strategy rec- 
ognition practice. In identification mini-games, students are shown 
a text and a corresponding self-explanation; they are prompted to 
select which of the previously learned strategies was used to gen- 
erate the example self-explanation. The extended practice game- 
based menu also allows students to monitor their performance 
within the system by accessing achievement screens. These pop- 
up screens allow students to view how they have performed in 


My Character iSTART Level 
23 -- Serious Alien Intelligence 
You have 30 assigned text(s) bis 
Next text is "Water, Carbon, and Nitrogen Cycles” 
You must complete all assignments before spending iBucks 
Earn more iBucks: 
008Test080808 [Showdown | 


iBucks (i8) 
30719 


both the identification mini-games and within the generative prac- 
tice games. Finally, students can also choose to personalize the sys- 
tem interface by changing the background color or editing an 
avatar. These four types of game-based features (generative prac- 
tice, identification mini-games, achievement screens, and person- 
alizable features) provide students with a substantial amount of 
control and personalization over their experience within iSTART- 
ME. 


1.2. Current study 


iSTART-ME has been effective at sufficiently motivating stu- 
dents while maintaining effective self-explanation instruction 
(Jackson, Davis, Graesser, & McNamara, 2011; Jackson & 
McNamara, 2013). However, this system provides students with 
an opportunity to exert high amounts of agency over their learning 
experience. Thus, iSTART-ME affords researchers a unique oppor- 
tunity to examine how students control and regulate their experi- 
ences by investigating how they choose to interact with various 
system features and what factors are mitigating these interactions. 
Previous work has shown that students’ ability levels are highly 
related to their ability to self-regulate (Kitsantas et al., 2008; 
McClelland et al., 2007; Zimmerman et al., 1992). This study builds 
off of this work while also taking advantage of the unique design of 
the iSTART-ME interface to examine how individual differences in 
reading ability influence students’ behaviors (i.e., patterns of inter- 
actions) and subsequent learning outcomes. The ultimate goal of 
iSTART-ME is to improve reading comprehension ability; thus, 
understanding how the effects of this intervention depend on prior 
ability is crucial in evaluating the effectiveness of the system. To 
this end, this study investigates the extent to which prior reading 
ability is a mitigating factor in students’ ability to control their 
behaviors during a learning task. In this study there are four pri- 
mary questions. 


(1) Do differences in target skill performance between high and 
low reading ability students emerge over time? 

(2) Do dynamic analyses successfully capture variations in stu- 
dents’ choice patterns with game-based features as they 
manifest across multiple training sessions? 

(3) How do students’ interaction patterns vary as a function of 
individual differences in prior reading ability level? 

(4) Do variations in students’ choice patterns associated with 
reading ability emerge over time? 


My Skills 
wit 


Spend my iBucks to: 


Change the Interface Cost 
Change background color 300 if 
Swop out the tutor 300 ifs 


Edit my character 


Cmcbiiagey See 


Fig. 1. Screenshot of iSTART-ME selection menu. 
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A unique contribution of this study stems from the dynamic 
analysis of log data to investigate which game-based features stu- 
dents interact with and how. Our goal is to provide researchers 
with deeper understanding of how students’ ability to exert control 
and regulate themselves emerges and manifest across time. 


2. Method 
2.1. Participants 


This study included 40 high-school students (50% male; mean 
grade level of 10.4; mean age of 15.5 years; 17% were Caucasian, 
73% were African-American, and 10% reported other nationalities), 
from a mid-south urban environment. The sample included in this 
study is a subset of 124 students who participated in a larger effi- 
cacy study that compared learning gains across three conditions: 
iSTART-ME, iSTART-Regular, and a no-tutoring control (for more 
information see Jackson & McNamara, 2013). The current work 
focuses solely on the students who were assigned to the iSTART- 
ME condition, as they were the ones who had access to the full 
game-based system. 


2.2. Procedure 


Students completed an 11 session experiment consisting of a 
pretest, eight training sessions, a posttest, and a delayed retention 
test. During the first session, students answered a battery of ques- 
tions. Particularly relevant for the current work, this pretest 
included an assessment of prior self-explanation (SE) ability and 
a standard measure of reading comprehension (Gates-MacGinitie 
Reading Test; MacGinitie & MacGinitie, 1989). During the subse- 
quent eight sessions, participants interacted with the iSTART-ME 
system, which included the introduction, demonstration, practice 
and extended practice modules. Each training session lasted 
approximately 1 h; during this time students were free to interact 
with any feature in the extended practice interface, which included 
generative practice, identification mini-games, personalizable fea- 
tures, and achievement screens. The data reported in this study 
primarily focus on students’ interactions with these game-based 
features during extended practice. These eight training session 
were originally designed to compare the effects of games versus 
non-games over long-periods of time (Jackson & McNamara, 
2013); however, the current work utilizes the log data from 
iSTART-ME to begin to understand how students control and regu- 
late their behaviors during learning tasks. During the 10th session, 
students completed a posttest, which was included a measure of 
self-explanation ability similar to the one in the pretest. One week 
after the posttest, students completed a retention test with mea- 
sures assessing long-term self-explanation ability. 


2.3. Materials and measures 


2.3.1, Reading comprehension 

Students’ reading comprehension was assessed using the Gates- 
MacGinitie Reading Test (MacGinitie & MacGinitie, 1989). This test 
included 48 questions designed to assess the general reading abil- 
ity of each student. In this task students were asked to read a pas- 
sage of text and then answer two to six comprehension questions 
about the material in the passage. Students were given 20 min to 
complete the test. This test is a well-established measure that pro- 
vides information about students’ reading comprehension ability 
(a = .85-.92, Phillips, Norris, Osmond, & Maynard, 2002). 


2.3.2. Self-explanation ability 

During the pretest, posttest, and retention students completed a 
self-explanation task, which was scored on a scale of 0-3, using the 
previously described iSTART algorithm scoring system. Students 
read through one of three science texts and provided self-explana- 
tions for specified target sentences (for a total of eight self-expla- 
nations per text). Self-explanation scores were averaged to 
provide a measure of students’ ability at each testing time. Stu- 
dents’ self-explanations were also evaluated during the training 
portion of the study and scored using the iSTART algorithm. 


2.3.3. Interaction categories 

Students’ interactions in iSTART-ME were logged within the 
iSTART-ME database. This process data was then organized accord- 
ing to the function afforded by each feature: generative practice, 
identification mini-games, personalizable features, and achieve- 
ments screens. 


2.3.3.1. Generative practice. Within iSTART-ME, students have the 
opportunity to engage in practice that requires them to generate 
their own self-explanations for a presented text (center box in 
Fig. 1). An activity was categorized as generative practice when 
students chose to engage with any of the three practice environ- 
ments: Coached Practice, Map Conquest, and Showdown. 


2.3.3.2. Identification mini-games. In addition to generative practice, 
six mini-games provide a different form of practice that targets the 
iSTART-ME strategies (bottom-right column of Fig. 1). As described 
earlier, these games present a text with an example self-explana- 
tion, requiring students to identify which strategies were used 
within the example provided. 


2.3.3.3. Personalizable features. Within iSTART-ME, students have 
the opportunity to personalize features within their environment, 
including editing an avatar, customizing the background theme, 
or changing their pedagogical agent (bottom-left options in Fig. 1). 


2.3.3.4. Achievement screens. As students progress through the 
iSTART-ME system they have the opportunity to earn points, win 
trophies, and advance to higher skill levels. These options are avail- 
able throughout the system as mouse-overs or pop-up screens. For 
example, students can click on “My Trophies” (right-side of Fig. 1) 
to view a record of their trophy achievements across all practice 
environments within iSTART-ME. 


3. Quantatative methods 


The iSTART-ME system captures and records all interactions 
that students perform within the extended practice selection 
menu. This study utilizes time-stamped process data to chronolog- 
ically categorize each student’s choices across the multiple training 
sessions. Overall, there were 11,343 interactions logged with each 
student making an average of 284 interactions across all 8 training 
sessions. Leveraging this extensive dataset, the probability of each 
student’s set of interactions and the amount of order (i.e., Entropy) 
that they exerted while engaging with these game-based features 
was Calculated. These methodologies are of particular importance 
to understanding the emergence of patterns within students’ 
choices as they both capture dynamic movements (or changes) 
that are often missed by more traditional (i.e., static) measures. 
This study is one of the first to use of both probability and Entropy 
analyses to capture nuanced patterns that emerge within students’ 
choice patterns. 
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3.1. Conditional probabilities 


Students’ probabilities of interactions were calculated using a 
statistical sequencing procedure similar to that used in D’Mello, 
Taylor, and Graesser (2007). This calculation can be described as 
LUI; > X;41], where L is the likelihood function of the student’s cur- 
rent interaction (J) at specific time point t, and X is the next inter- 
action at the next time point (t + 1). More simply, we calculated the 
probability of a student’s interaction with an interface feature 
given their previous interaction. For instance, if Jane interacts with 
a generative practice game, we will use the above formula to 
examine what feature Jane is most likely to interact with next 
(e.g., another generative practice, an identification mini-game, a 
personalizable feature, or an achievement screen). These likelihood 
probabilities were calculated for each interaction transition yield- 
ing a unique pattern for each student. 


3.2. Entropy 


Students’ propensity to engage with game-based features 
within iSTART-ME in an ordered fashion was calculated using 
Shannon Entropy (Shannon, 1951). Entropy is a statistical measure 
derived from thermodynamics that classifies the order of a time 
series (Clausius, 1865). This measure classifies the probability of 
uncertainty (e.g. order) in a system (see Formula 1; pi is the prob- 
ability of a given state). Entropy has previously been used in psy- 
chology to investigate patterns of order in students’ choices 
(Fasolo, Hertwig, Huber, & Ludwig, 2009; Grossman, 1953; Snow, 
Jacovina, et al., 2014). For instance, Snow and colleagues used 
Entropy analysis to quantify how students’ behaviors varied (i.e., 
were random or controlled) during game-play. These analyses pro- 
vided a fine-grained metric of ordered behavior that would have 
otherwise been missed. Indeed, Entropy provides a deeper under- 
standing of how students choose to regulate their interactions by 
examining the amount of order exhibited in their choice of interac- 
tions. In general, low Entropy suggests highly organized choice 
patterns, whereas high Entropy suggests disorganized choice pat- 
terns. Within the context of this study, Entropy scores of 0 would 
be indicative of perfect order. This means that a pattern consists 
of the exact same interaction over and over again. For instance, if 
Joe was to select an identification mini-game 10 times and he 
chooses nothing else, his Entropy score would be 0. Conversely, 
within the context of this study the highest Entropy score a student 
can earn is 2, which would be indicative of a disordered selection 
pattern. Thus, through the use of an Entropy analysis we can cap- 
ture the amount of order (or disorder) present in students’ choice 
pattern. 

N 
H(x) = —} > P(xi)(log.P(xi)) (1) 
i=0 


L 


4. Results 


This study examines how differences in students’ ability to con- 
trol their behaviors (i.e., interaction choices within isTART-ME) 
relates to their target skill acquisition (i.e., daily self-explanation 
quality) and varies as a function of prior reading ability. First, using 
repeated measures analysis of variance, we examine differences in 
learning outcomes between high and low reading ability students. 
Conditional probability and Entropy analyses are then calculated 
on students’ log data to investigate how they interacted within 
the iSTART-ME system. Finally, we investigate how high and low 
reading ability students vary in their use (i.e., probability of inter- 
action) and control (i.e., Entropy) of various game-based features 
embedded within iSTART-ME. 


4.1. Learning outcomes 


Prior research has examined how students, of varying abilities, 
performed differently during interactions within the iSTART 
system (Jackson, Boonthum, & McNamara, 2010; Jackson, 
Boonthum-Denecke, & McNamara, 2012; Jackson, Varner, 
Boonthum-Denecke, & McNamara, 2013). Specifically, results from 
these studies have revealed that students with low prior reading 
ability improve performance across time, catch up, and become 
indistinguishable from high reading ability students in terms of 
performance. Hence, one goal of this study is to examine how indi- 
vidual differences in reading ability are related to changes in stu- 
dents’ self-explanation quality across time. A repeated-measures 
ANOVA revealed a significant main effect for testing time, indicat- 
ing that students significantly improved their self-explanation 
quality scores from pretest (M=1.52, SD=.62) to posttest 
(M = 2.11, SD=.63), F(1,38) = 26.05, p<.001. To investigate the 
impact of individual differences on self-explanation quality, a med- 
ian split was conducted on students’ pretest reading comprehen- 
sion scores (creating high and low ability groups of students). An 
ANOVA using the median split on reading comprehension indi- 
cated that low ability students (M = 1.35, SD =.61) generated sig- 
nificantly lower quality self-explanations at pretest than high 
ability students (M = 1.78, SD = .55), F(1,38) = 5.13, p < .05. In con- 
trast, an ANOVA on the posttest scores revealed that the self-expla- 
nation quality of students with a low prior reading ability 
(M = 2.11, SD = .68) was not significantly different from high ability 
students (M = 2.12, SD =.57; F(1,38) = 0.01, p = .96). A mixed factor 
repeated-measures ANOVA, including the median split with the 
pretest and posttest self-explanation scores, yielded a marginally 
significant interaction between testing time and students’ reading 
ability, F(1,38) =3.73, p=.06. These results reveal that both high 
and low ability students improved their self-explanation skills 
from pretest to posttest and interestingly, by the end of the 8 train- 
ing sessions, low ability students were able to match the posttest 
outcomes of high ability students. 

To examine how the differences in self-explanation quality 
emerge over time, analyses were performed to examine students’ 
self-explanation quality scores during training (see Fig. 2). A 
mixed-factors ANOVA on students’ self-explanation scores during 
training indicated a significant linear interaction between reading 
ability and training session, F(1,29) = 5.43, p< .05. Thus, students 
with low prior reading ability tended to improve self-explanation 
scores across sessions and over time performed more similarly to 
students with high reading ability. Bonferroni-adjusted pairwise 
comparisons indicated that high ability students produced signifi- 
cantly better self-explanation scores than low ability students for 
Days 1 (t=5.89, p*dusted< 01), 2 (t=3.96, prdusted< 01), 3 
(t=3.89, prdusted< 91), 4 (t=2.27, p2dusted< 95), 5 (t=2.87, 
padusted < 91), and 7 (t= 2.74, p*dusted < 95). In contrast, adjusted 
pairwise comparisons indicated that low and high ability students 
did not produce significantly different self-explanation scores dur- 
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Fig. 2. Self-explanation performance during training. 
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ing Days 6 (t= 1.65, p*sted > 95) and 8 (t= 1.82, p2duste4 > 05), 
This trend suggests that students with low prior reading ability 
are able to improve their self-explanation quality, and by the end 
of training perform similar to the high reading ability students. 


4.2. Overall transitional probability patterns 


One explanation for why the learning trends observed in this 
study may emerge relates to the assumption that the low ability 
students’ control and regulatory behaviors evolve while they 
engage within iSTART-ME. Indeed, previous work has shown that 
students’ ability to regulate and control their behaviors develops 
across time (Eshel & Kohavi, 2003). Thus, a question inherent to 
this study regards the patterns of interactions within iSTART-ME 
and how those differ as a function of students’ prior ability. To 
investigate how individual differences in students’ pretest reading 
comprehension ability impacted the way in which students 
behaved and interacted within the game-based system, a transi- 
tional probability analysis was conducted (Fig. 3). This analysis 
revealed significantly different patterns between the two ability 
groups. Fig. 3 provides a visual display of the transition probabili- 
ties for each group (high ability in black and low ability in purple), 
with numbers inside a box representing the likelihood of selecting 
the same feature again, and numbers near a line indicating the 
likelihood of transitioning from one feature to another. Overall, 
students spent the majority of their time interacting with and tran- 
sitioning between generative practice games and identification 
mini-games (high ability 76%; low ability 72%). By examining the 
trends for each ability level, patterns indicate that low ability stu- 
dents were significantly more likely to interact with the generative 
practice games, F(1,38)=6.23, p<.05, whereas high ability stu- 
dents were more likely to interact with the identification mini- 
games F(1,38) = 16.62, p< .001 (see asterisks in Fig. 3 for signifi- 
cantly different patterns). Low ability students were also more 
likely to check their personal achievement screens before, 
F(1,38) = 6.39, p <.05, and after F(1,38) =6.09, p< .05, they inter- 
acted with a generative practice game. Thus, low ability students 
seem to be monitoring their progress through the system (via 
achievement screens). These patterns indicate that students’ abili- 
ties affect how they interact with the computer-based learning 
environment in a variety of ways. 


4,3. Interactions choices across time 


The overall transitional probability analyses provide some 
insight into how students’ patterns of interactions within a 
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Fig. 3. Transition probabilities for high and low reading ability students. p < .05* 


game-based ITS varied as a function of individual differences in 
reading ability. However, the previous analyses provide little infor- 
mation about how these differences manifest overtime. Using the 
median split on Gates reading comprehension, interaction proba- 
bilities were calculated separately across all training sessions 
(see Fig. 4). Most students spent the first session going through 
the introduction, demonstration, and part of regular coached prac- 
tice (i.e., they did not progress into the selection menu on training 
sessions one). However, the majority students began to interact 
with the selection menu during the second session. Thus, Fig. 4 dis- 
plays the distribution differences between the low and high ability 
readers for session 2-8. During the first session with the selection 
menu (Day 2 in Fig. 4), all students use the system features in sim- 
ilar patterns (i.e., there are no significant differences in the proba- 
bility distribution). However, high ability students chose to 
interact with the identification mini-games significantly more 
often than did the low ability students on Days 3 
(F(1,38)= 10.86, p<.001), 4 (F(1,38)=12.35, p<.001), and 5 
(F(1,38) = 15.02, p<.001). Conversely, the low ability students 
chose to engage with generative practice significantly more often 
than the high ability students on Days 3 (F(1,38) = 5.73, p <.05), 
4 (F(1,38) = 13.58, p < .001), and 5 (F(1,38) = 13.52, p < .001). Inter- 
estingly, the identification mini-game probability transitions 
between low and high ability students converge across training 
and are no longer significantly different by training Days 6 
(F(1,38)=3.78, p=.06), 7 (F(1,37)=1.19, p=.20), and 8 
(F(1,29) = 1.41, p=.24). Similarly, the differences between high 
and low ability students in probability of interactions with gener- 
ative practice games converge on Day 6 (F(1,38) = 2.20, p=.15), 7 
(F(1,37) = 1.17, p =.20), and 8 (F(1,28) =.69, p =.41). 

The results illustrated in Fig. 4 indicate that high ability stu- 
dents quickly settle into a pattern that is consistent across Days 
(3-8) and predominantly focus on the identification mini-games 
(roughly 40-50% of all choices). By contrast, the low ability stu- 
dents initially focus on the generative practice (which provides 
them with more scaffolding and feedback), and later transition into 
a pattern almost identical to high ability users. These results sug- 
gest that low ability students are regulating their learning (as also 
suggested in Fig. 3) and adapting their system choices based on the 
level of their performance. 


4.4, Measure of order in interaction patterns across time 


To investigate how students of varying reading ability regulated 
and controlled their behaviors across multiple sessions, an Entropy 
analysis was conducted. Using the median split on Gates reading 
comprehension, Entropy was calculated separately for each stu- 
dent across training Days 2-8 (see Fig. 5). During the first session 
with the selection menu (Day 2 in Fig. 5), low ability students’ 
interactions patterns demonstrated higher levels of Entropy com- 
pared to high ability students F(1,38)= 4.507, p<.05. Thus, low 
ability students demonstrated a more disorganized interaction pat- 
tern compared to high ability students during the first session 
within the interface. During Days 3-6, this variation in order disap- 
pears as students of both high and low ability show comparable 
levels of Entropy (i.e., order) within their choice patterns. Interest- 
ingly, on Day 7, low ability students again show significantly more 
disorder with their choice patterns compared to high ability stu- 
dents, F(1,38) = 4.47, p < .05. However, on Day 8 there are no differ- 
ences in amount of demonstrated order between high and low 
ability students. Interestingly, the days when low ability students 
show more disorder in their choice patterns correspond to days 
when these students are interacting with a new feature predomi- 
nantly for the first time. On Day 2, the system is novel to all stu- 
dents; thus they may not know what to expect or how to 
effectively make choices within the system. Similarly, Entropy dif- 
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ferences on Day 7 correspond to the first day when low ability stu- 
dents switch their choice patterns to mirror the high ability stu- 
dents (i.e., identification mini-games). These results suggest that 
when low ability students begin interacting more frequently with 
novel features, they demonstrate a more disorganized choice pat- 
tern compared to high ability students. However, this disorganiza- 
tion dissipates over time. 


5. Discussion 


ITSs often offer students a high level agency as a means to cre- 
ate a personalized learning experience. These experiences have 
been found to vary as a function of various individual differences 
(Baker et al., 2008; Snow et al., 2013b; Snow, Allen, Jackson, & 
McNamara, 2014). This study examines how students control and 
regulate their behaviors by investigating how they choose to inter- 
act with specific game-based features and why they chose to 
engage with them. Using two novel methodologies, conditional 
probability and Entropy analyses, we attempt to gain a deeper 
understanding of how various system trajectories form and mani- 
fest overtime. The current work revealed differences in behavior 
patterns (and performance) between students of low and high 
prior reading ability. These results provide computational signals 
of how students vary in their ability to regulate their behaviors 
across time, and potentially catch up to their more skilled peers. 

Over the course of training within iSTART-ME, students with 
low prior reading ability were able to catch up and match the per- 
formance of high ability students (right-side of Fig. 2). This finding 
replicates previous work with the original version of iSTART, which 
found that over time, low ability students were able to improve 


performance and match the performance of high ability students 
(Jackson et al., 2010, 2013). To examine what could be contributing 
to the merging of these performance differences we investigated 
how students chose to regulate and control their behaviors while 
engaging with the iSTART-ME interface. 

The overall transitional probability analyses revealed that high 
ability students choose to engage with the identification mini- 
games significantly more often than the low ability students 
(Fig. 3). In contrast, low ability students interact significantly more 
with the generative practice environments. Interestingly, the low 
ability students tend to monitor their system progress while inter- 
acting with generative practice games, as evidenced by a signifi- 
cantly higher rate of transitions to and from achievement 
screens. Thus, it appears that the low ability students are tracking 
their progress through the computer-based learning environment, 
and may also be adapting their behavior accordingly. This finding 
is contradictory to previous work that has found that high ability 
students are more likely to monitor and observe their behavior 
during learning tasks (Schunk, 1983, 2008). However, in this study 
students were engrossed in a game-based environment where they 
received varying levels of feedback based on the interactions they 
chose. Thus, as low ability students interacted more frequently 
with generative practice games (where they received a relatively 
steady dose of feedback), they may have been more aware of their 
performance within the system. Indeed, this relatively steady 
stream of feedback may have made the students more aware of 
their progress and inadvertently prompted them to monitor their 
achievements more frequently. 

To examine emergent behavior patterns between high and low 
reading ability students a separate probability analyses across 
training sessions was also conducted. The distributions repre- 
sented within Fig. 4 suggest that during the first few sessions, 
low ability students choose to interact most frequently with the 
generative practice environments. In contrast, students with a high 
reading ability tend to engage with the various identification mini- 
games (approximately half of the time). Examining the trends in 
Fig. 4, it is also evident that the low ability students adapt their 
interaction patterns across time (i.e., the distribution of choices 
changes from Day 3 through 7), and ultimately mirror the behav- 
iors of high ability students. 

The probability analyses demonstrated differences in which 
game-based features students chose to interact. While informative, 
these analyses alone cannot capture whether or not there is order 
present in those interaction patterns. Thus, using an Entropy anal- 
ysis, we were able to distinguish differences in the way in which 
low and high ability demonstrated controlled patterns of interac- 
tion. The distributions represented within Fig. 5 suggest that dur- 
ing the first session, low ability students interacted in a more 
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disordered fashion compared to students with a high reading abil- 
ity. However, this difference disappears between Days 3 and 6, 
where both groups of students engage in an ordered fashion. 
Although high and low ability students interact with different fea- 
tures, both groups demonstrate controlled and ordered patterns of 
interactions. This is a characteristic of SRL and thus suggests that 
both high and low ability students regulate their behaviors, albeit 
in different ways. On Day 7, we observed, again, that low ability 
students revealed more disorder in their interaction patterns. 
Interestingly this resurgence of disorder coincides with the first 
day that low ability students began interacting with identification 
mini-games at a comparable rate to the high ability students. Sub- 
sequently, when low ability students changed their interactions 
they also generated significantly lower self-explanation quality 
scores compared to the high ability student. These results support 
the notion that with time and practice, low ability students can 
effectively self-regulate their interactions within learning environ- 
ments, particularly when these environments are game-based. 

These exploratory analyses begin to reveal computational sig- 
nals of catching up that vary as a function of individual differences 
in reading ability and thus, impact corresponding learning out- 
comes. However, more research on individual differences and their 
impact on students’ interactions with system features are still 
needed. Specifically, it is useful to understand how students’ prior 
experience with technology or video games influences the ways in 
which they regulate and control their behaviors within ITSs. 
Expanding this type of work will allow researchers to improve 
the designs of learning environments that adapt to students’ 
strengths and preferences. Specifically, future research should 
identify optimal and non-optimal behaviors; examine how to rec- 
ognize these patterns in real-time, and design effective methods 
for adapting the system to match students’ needs. 

One limitation of this study is that no traditional self-report 
measure of SRL used. Traditionally, researchers have used self- 
reports to assess students’ regulatory abilities. However, in the cur- 
rent work we relied upon log data analysis to capture behaviors as 
they occurred and evolved during the learning task. In the future, 
confirmatory studies demonstrating concurrent validity are 
needed to examine how the dynamic analysis of log-data relates 
to traditional measures of SRL behaviors. 

The findings presented in this paper are some of the first to 
trace students’ interaction patterns and subsequent learning gains 
across time at both coarse-grained and fine-grained levels. Conse- 
quently, these results reveal how students’ interaction patterns 
manifested and emerged over time while also varying as a function 
of individual differences in reading ability. These new and innova- 
tive methods afford researchers the opportunity to assess varia- 
tions in students’ behaviors overtime and their ultimate impact 
on targeted skills. These results provide valuable insight into how 
users interact with complex environments through various behav- 
iors, as well as why these different patterns emerge. 
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